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Equation of the Line 

Regression Coefficients 

Matrix Approach 

The Least Squares 

The Coefficient of Determination 


Assumptions of the Regression 


+ several explanatory variables to 
predict an outcome 


- an extension of linear (OLS) 
regression modeling 


^ continuous outcome variable 
o continuous or categorical predictors 





Weight Horsepower 





Simple 
Linear ys Do + DX 


Regression 





Dependent variable (DV) Independent variables (IVs) 
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y = 1.23 + .442 X, + .265 Χο + «045 X3 


- — SSR SSE 
Coefficient of Deternination > R2 = —— = 1 — — 
SST SST 

Sum of Squares Total > SST = Σο — y)? 


Sum of Squares Regression > SSR = Σο = y? 


Sum of Squares Error > SSE = XO = y)? 


e R? - variance of the dependent variable explained by the regression model 
e Perfect fit, SSE = 0, and R? is 1; total failure, SSE = SST, R? = 0 


OLS Regression Results 


Dep. Variable: housing price index R-squared: 0.980 
Model: OLS Adj. R-squared: 0.974 
Method: Least Squares F-statistic: 168.5 
Date: Fri, 13 Apr 2018 Prob (F-statistic): 7.32e-14 
Time: 16:31:58 Log-Likelihood: -55.164 
No. Observations: 23 AIC: 122.3 
Df Residuals: 17 BIC: 129.1 
Df Model: 5 
Covariance Type: nonrobust 


b; bi 


bx = — = 
PD Sebi) SSDT 


Significance test > 


coef >It} [9025 0.975] 


Intercept -389.2234 187.252 -2. 0.053 -784.291 5.844 
total unemployed -0.1727 2.399 -0.072 0.943 -5.234 4.889 
long interest rate 5.4326 1.524 3.564 0.002 2.216 8.649 


Standard Error of Estimation > 5. = 
e federal funds rate 32.3750 9.231 3.507 0.003 12.898 51.852 





consumer price index 0.7785 0.360 2.164 0.045 0.020 1.537 
gross domestic product 0.0252 0.010 2.472 0.024 0.004 0.047 


Omnibus: 1.363 Durbin-Watson: 1.899 
Prob(Omnibus): 0.506 Jarque-Bera (JB): 1.043 
Skew: -0.271 Prob(JB): 0.594 

Kurtosis: 2.109 Cond. No. 4.58e+06 











Categorical to Continuous 


Label Binary 
Encoding Encoding 


Continuous to Categorical 


Unsupervised Supervised Binning 


Binning 










Ordinal One Hot 
Encoding Encoding 
Equal wrath icc = 
Binning Binning Binning 


Frequency Target Mean 
Encoding Encoding 





Data collection 


Transform data into a 
stationary series 


Building a linear 


τ — Examine the linear 
regression model for ———® Parameters estimation ———> 


dan model 
^ drop insignificant coefficients 
o reduce the average error of predictions " ' 
model adequate? 
^ estimation of regression coefficients unstable 
o due to multicollinearity ves | 
The linear 
model 


- a simpler model is a better model 
o more insight into the influence of predictors 
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Grade 





Grade. 


Grade, 


Grade, 
0) 
0) 


In [40]: Location_County = Sales.County # use df from result 
dummy County= pd.get_dummies(Location_County) 


dummy County 
"A 


Out[40]: 





Adair Adams Allamakee Appanoose Audubon Benton a Boone Bremer Buchanan ... Wapello Warren Washington Wayne Webster Winnel 
0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 
1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
2 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 
3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
5 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
6 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 
7 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
9 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 


In [41]: County Sales - pd.concat([Sales, dummy County ], axis-1) £ combining sales data with the location(county) data 
County Sales.shape 


Out[41]: (269258, 103) 


Original Data 
10 





StandardScaler 
© 33 





-2.0-1.5-1 


MinMaxScaler 


14 





Assumptions of regression 





The text lists four assumptions for multiple regression: 


> 1. The relationship is linear 

> 2. The errors have the same variance 

3. The errors are independent of each other 
> 4. The errors are normally distributed 
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- (Σχκθίαιν) -(Σχιχρίξχενλ) — 755175-(-375)*8.5 





1 (Ex2)(Exi)-(Exyj)) 9.875 * 7.5 — (—3.75) + (-375) 


~ _(2xf)@x2y)- (Zx1x2)(Zx1y) _ 9.875 + 8.5 — (—3.75)+ 1.75 





(2 xf) (Z x3) — (Lx 4x2)? 9.875 * 7.5 — (—3.75) * (—3.75) 


The estimated regression line would be 


= 31.37+0.75p+1.5q 
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Model ml.summary() - ml.score() -prediction 
Evaluation p-values, Cl; SE, beta accuracy 
ml.params() 


Preferred Use Statistical inference; Accurate prediction 
hypothesis testing 

Regularization No regularization by L2 by default, ο. 
default (penalty = ‘none’) 

Hyperparameter | User created GridSearchCV 

Tuning 





import statsmodels.api as sm 
X = sm.add constant(X) 


# Note the swap of X and y 
model = sm.OLS(y, X).fit() 


# Statsmodels gives R-like statistical output 
model. summary ( ) 


OLS Regression Results 


Dep. Variable: _| petal_width 0.927 





Wed, 05 Apr 2017 | Prob (F-statistic): | 4.68e-86 


Tim 21:56:10 Log-Likelihood: | 24.796 
No. Observations: | 150 AIC: -45.59 


www p | — 
Covariance we emma (| | 


C em seem Pi [so conc 
πας [ooo [ss ooo] 042-0265 | 


Omnibus: 5.765 | Durbin-Watson: 
Prob(Omnibus): |0.056 | Jarque-Bera (JB): 
see σπα] 





Model OLS Adj. R-squared: 0.927 
Method: Least Squares F-statistic: 1882. 











coefficient 
of the x (IV) 








OLS Regression Results 


Model: OLS Adj. R-sguared: @.272 κ 








Method: Least Squares F-statistic: 5.213 
Date: Tue, 30 Jan 2018 Prob (F-statistic): 0.0473 R?: how well 
Time: 14:21:22  Log-Likelihood: -41.442 the model 
No. Observations: 12 A: 86.88 fits the data 
Df Residuals: 10 BIE: 87.85 
Df Model: 1 
Covariance Type: nonrobust 
coef std err t P»|t| [95.0% Conf. Int.] 

6.6364 20.546 8.597 0.000 130.858 222.415 
x 0.158 -2.261 -0.005 P-value: 
Se t E t Gr t t m t t tm GG RT E E D GG D GR GE UE EG GE GE D E SEU GGG GE GE GE CE E GER GE GE GL GE É GE UEGE GE GEGE GE —— signif. of x 
Omnibus: 1.934  Durbin-Watson: 1.182 (IV) for Y (DV) 
Prob(Omnibus): 0.380  Jarque-Bera (JB): 1.010 
Skew: -8.331  Prob(2B): 0.603 
Kurtosis: 1.742 Cond. No. 1.10e403 


DV:dependent variable/outcome/target; IV: Independent variable, factor, predictor, feature 


For every 1 unit change in x, the average change in y is b, 





-. coef stderr t P>|t| [95.0% Conf. Int. ] 
s a ασ n re Lee STE RUNE Cae ER CY MEE Lea Me 
const / 176.6364 20.546 0.597 0.000 130.858 222.415 
-8.3572 0.158 -2.261 0.047 -0.709 -0.005 
V o o o κκ. 
| P 
| / 
y=by+bx+e y = 176.63 + -0.357x + e 
X b 
y-intercept slope of N S 


(x=0) regression 





from sklearn import linear model as lm 


X = iris[["petal_length"]] 
y = iris["petal width" ] 


# Fit the linear model 
model = lm.LinearRegression| ) 
results = model.fit(X, y) 


# Print the coefficients 
print model.intercept_, model.coef_ 


-0.363075521319 [ 0.41575542] 


